Skip to content

ci: Add healthcheck for Github runners to run on a schedule#1030

Merged
terrykong merged 21 commits intomainfrom
chtruong/healthcheck
Sep 3, 2025
Merged

ci: Add healthcheck for Github runners to run on a schedule#1030
terrykong merged 21 commits intomainfrom
chtruong/healthcheck

Conversation

@chtruong814
Copy link
Contributor

What does this PR do ?

  • Add healthcheck for Github runners to run on a schedule (7 a.m. UTC)
  • Checks number of gpus and reboots if gpus less than expected
  • Re-runs health check if rebooted

Issues

List issues that this PR closes (syntax):

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

  • ...

Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@github-actions github-actions bot added the CI Relating to CI label Sep 1, 2025
@terrykong
Copy link
Collaborator

@ko3n1g to review

Copy link
Contributor

@ko3n1g ko3n1g left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did you test this a few times, is 3min enough? Otherwise looks like a clever extension of the original workflow

@terrykong terrykong added this pull request to the merge queue Sep 2, 2025
Merged via the queue into main with commit 4821ef8 Sep 3, 2025
21 checks passed
@terrykong terrykong deleted the chtruong/healthcheck branch September 3, 2025 11:37
terrykong pushed a commit that referenced this pull request Sep 6, 2025
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
guyueh1 pushed a commit to guyueh1/NeMo-RL that referenced this pull request Sep 15, 2025
PrinsYin pushed a commit to PrinsYin/RL that referenced this pull request Nov 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI Relating to CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants